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. IMAGE PROCESSING METHOD AND APPARATUS 

BACKGROUND OF THE INVENTION 

The present invention relates to an image processing method and 
"apparatus, and more particularly, to a method and apparatus for 
extracting an image, which meets a predetermined condition, from 
among a plurality of images. 

Medical images are helpful in diagnosing a patient condition. 
Medical images provide unique information depending on what modality 
is used to produce the medical images. Medical images produced by 
an appropriate modality are employed according to a purpose of 
diagnosis. 

In order to given more appropriate diagnosis on the basis of 
diverse information, a plurality of kinds of medical images produced 
by a plurality of modalities, such as, an X-ray computed tomography 
(CT) apparatus, a magnetic resonance imaging (MRI) apparatus, and a 
positron emission tomography (PET) apparatus or a single photon 
emission computed tomography (SPECT) apparatus are employed. 

For example, the X-ray CT apparatus and PET apparatus are used 
to image the same region of the same patient. Images produced by 
the X-ray CT apparatus and showing the structure of the cerebral 
parenchyma, and images produced by the PET apparatus and showing the 
active state of. the brain are used to diagnose a lesion in terms of 
the structure of the lesion and the function thereof. At this time, 
a synthetic image produced by superposing two different kinds of 
images in order to facilitate simultaneous grasping of the structure 
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and function. 

In order to produce such a synthetic image, two images are 
aligned with each other. The alignment may be referred to as 
registration. The registration is performed by a computer. 

For the registration, a technique described in, for example, 
"Multi-modal Volume Registration by Maximization of Mutual 
Information" written by W. M. Wells III et al. 
(http: / /splweb.bwh. harvard.edu: 8000/pages/papers/swells/mia- 
html/mia.html) is adopted.. 

Generally speaking, the registration is achieved through 
transformation through which the amount of mutual information and a 
candidate image v(x) shares with a reference L image u(x) is maximized. 



where T denotes the transformation from the apparatus of coordinates 



Namely, 



r^arg max I(u(x), v(T(x))) 

T 



(1) 



for the candidate image to the apparatus of coordinates for .the 



reference image. 



/(«(x), v(7X*))) 



(2) 



The formula 2 denotes mutual information. 



/(«(*), vCF(x))) = . 

• h(u(x)) + h(v(T(x))) - h(u(x), v(T(x))) 



(3) 



where h(-) denotes an entropy. 




(4) 



h(x,y) = - jp( x ,y)In p{x,y)dxdy 



(5) 



The above formulae provide an entropy. 
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The foregoing registration proves effective when it is 
previously apparent that two images result from visualization of the 
same region. Otherwise, even if transformation provides an image 
that shares the largest amount of mutual information with a 
reference image, the transformation does not always lead to 
significant registration. 

SUMMARY OF THE INVENTION 

Therefore, an object of the present invention is to provide a 
method and apparatus for extracting an image closest to a reference 
image from among a plurality of candidate images. 

(1) According to one aspect of the present invention, 
attempting to solve the aforesaid problem, there is provided an 
image processing method for: extracting a plurality of candidate 
images similar to a reference image from among a plurality of images 
by utilizing granulometry; transforming the plurality of candidate 
images on the basis of the reference image; calculating mutual 
information shared by each of the transformed candidate images and 
the reference image; and selecting a candidate image, which shares 
the largest amount of mutual information with the reference image, 
from among the plurality of candidate images. 

(2) According to another aspect of the present invention 
attempting to solve the aforesaid problem, there is provided an 
image processing apparatus comprising: a extracting means for 
extracting a plurality of ' candidate images similar to a reference 
image from among a plurality of images by utilizing granulometry; a 
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transforming means for transforming the plurality of candidate 
images on the basis of the reference image; a calculating means for 
calculating mutual information shared by each of the transformed 
candidate images and the reference image; and a selecting means for 
selecting a candidate image, which shares the largest amount of 
mutual information with the reference image, from among the 
plurality of candidate images. 

According to the aspects (1) and (2) of the present invention, 
a plurality of candidate images similar to a reference image are 
extracted from among a plurality of images by utilizing granulometry . 
The candidate images are transformed based on the reference image . 
Mutual information shared by each of the transformed candidate 
images and the reference image is calculated. A candidate image 
that shares the largest amount of mutual information with the 
reference image is selected from among the plurality of candidate 
images. Consequently, an image closest to the reference image can 
be extracted from among the plurality of candidate images. 

Preferably, the transformation includes matching of 
magnifications so that the transformation can be achieved 
effectively. 

Preferably, the transformation includes alignment of 
barycenters so that the transformation can be achieved effectively. 

Preferably, both the reference image and candidate images are 
medical images so that diagnosis can be achieved based on a 
synthetic image. 

Therefore, the present invention provides a method and a 
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apparatus capable of extracting an image closest to a reference 
image from among a plurality of candidate images . 

Further objects and advantages of the present invention will be 
apparent from the following description of the preferred embodiments 
of the invention as illustrated in the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of a apparatus of an example of an 
embodiment of the present invention. 

Fig. 2 is a flowchart describing actions to be performed by the 
apparatus of the example of the embodiment of the present invention. 

Fig. 3 is a conceptual diagram showing image frames. 

Fig. 4 is a conceptual diagram showing image frames. 

Fig. 5 is a conceptual diagram showing image frames. 

Fig. 6 shows coordinates representing barycenters in images. 

Fig. 7 is a conceptual diagram showing an image frame. 

Fig. 8 is a functional block diagram showing the apparatus of 
the example of the embodiment of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

Referring to drawings, an embodiment of the present invention 
will be described below. Fig. 1 is a block diagram of an image 
processing apparatus. The present apparatus is an example of an 
embodiment of the present invention. The configuration of the 
present apparatus presents an example of an embodiment of an image 
processing apparatus in which the present invention is implemented. 
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Actions to be performed by the present apparatus present an example 
of an embodiment of an image processing method in which the present 
invention is implemented. 

As shown in Fig. 1, the present apparatus includes a computer 
100. The computer 100 receives images tt> be processed. The 
computer 100 includes a memory 102 in which the received images are 
stored. Various kinds of data and various programs that are used by 
the computer 100 are stored in the memory 102. When the computer 
100 runs the programs stored in the memory 102, various procedures 
of image data processing are carried out. 

The computer 100 also includes a display unit 104 and an 
operating unit 106. An image and other information provided by the 
computer 100 are displayed on the display unit 104. A user handles 
the operating unit 106, and various instructions and information are 
transferred from the operating unit 106 to the computer 100. The 
user uses the display unit 104 and operating unit 106 to operate the 
image processing apparatus interactively. 

Actions to be performed by the present apparatus will be 
described below. Fig. 2 is a flowchart describing the actions. The 
actions of the present apparatus are performed when the computer 100 
runs a program stored in the memory 102. 

As illustrated, at step 202, a reference image is read. 
Consequent ly, for example, an image frame containing an image A 
shown in Fig.' 3(a) is stored in the memory 102. The image A is, for 
example, a tomographic image of the head produced by a PET apparatus 
The image A is a halftone image showing the function of the brain. 
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At step 204, received images are read. Consequently, for 
example, an image frame containing an image B shown in Fig. 3(b). is 
stored in the memory 102. The image B is, for example, a 
tomographic image of the head produced by an X-ray CT apparatus. 
The image B is a halftone image showing the structure of the 
cerebral parenchyma. 

A plurality of images preserved in an external image file or 
the like are read. The plural ' images are different from one another 
in terms of, for example, a position of slicing. 

At step 206, candidate images similar to the reference image 
are extracted from among the received images. For the extraction of 
the candidate images, granulometry that is a kind of morphological 
operation is utilized. 

The granulometry is known as described in, for example, 
"Granulometry; Mathematical Morphology in Statistical Sense" 
(http: //kuva .mis . hiroshima~u . ac . jp/-asano/Kougi/00a/Inf staatll/11- 
10.pdf ) . 

The granulometry is adapted to the reference image and the 
plurality of received images. As a result of the granulometry, a 
pattern spectrum is provided for each of the reference images and 
plurality of received images. The similarity between the pattern 
spectrum of each received image and the pattern spectrum of the 
reference image is checked. Received images whose pattern spectra 
are similar to the pattern spectrum of the reference image are 
extracted as candidate images . Thus, only the received images whose 
pattern spectra are similar to the pattern spectrum of the reference 
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image are extracted in advance, whereby succeeding image processing, 
is performed efficiently. 

At step 208, the candidate images are transformed. The 
transformation is geometric transformation from the apparatus of 
coordinates for the candidate images to the apparatus of coordinates 
for the reference image. The transformation includes matching of 
magnifications and alignment of ■ barycenters. 

The matching of magnifications is the processing of matching 
the size of a candidate image with the size of the reference image 
as described below. First, the reference image and candidate image 
are binary-coded. The binary coding is such that as for the 
reference image that is the image A, among all pixels, that 
constitute data of the image frame containing the image A, pixels 
whose values are egual to or larger than a predetermined threshold 
are set to Is and pixels whose values fall below the threshold are 
set to 0s. As for the candidate image, among all pixels that 
constitute data of the image frame containing the image B, pixels 
whose values are equal to or larger than a predetermined . threshold 
are set to Is, and pixels whose values fall below the threshold are 
set to 0s. 

This results in, for example, image frames shown in Fig. 4(a) 
and Fig. 4(b). In the image frames, all the pixels constituting the 
.data of the images a and b are set to Is, and pixels constituting 
the data of the other portions are all set to 0s* 

In general, the moment of an image is expressed as follows: 
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M(p 9 q)= J^i p j q (6) 

A magnification required to match the size of the image b with the 
size of the image a is given as follows: 

M(0,0)Z> 

where M(0,0) denotes the 0-order moment. The magnification k is 
calculated for each of the plurality of candidate images. The 
magnifications k are adapted to the candidate images , whereby the 
candidate images have the same size as the reference image. 

In order to align barycenters with each other, the barycenters 
of the images a and b in the respective image frames are calculated. 
The barycentric coordinates (m,n) are calculated as follows: 

m = M(l,0)/M(0,0) (8) 

n = M(0,l)/M(0,0) (9) 
where M(1,0) and M(0,1) denote first-order moments. 

The above formulae are adapted to the image frames shown in Fig. 
4(a) and Fig. 4(b), whereby the coordinates representing the 
barycenters. aO and bO of the images a and b are determined as shown 
in Fig. 5(a) and Fig. 5(b). The barycenters are not only the 
barycenters of the images a and b but also the barycenters of the 4 
images A and B. 

Assuming that the coordinates representing the barycenters aO 
and bO of the images A and B are, for example, (iO,jO) and (iO^jO 1 ) 
as shown in Fig. 6, a magnitude of correction for the position of a 
barycenter is given as follows: 
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A/ = / 0 -V ' (10) 

*j = Jo-Jo (ID 
where Ai denotes the magnitude of correction for the position of a 
barycenter in the direction of an I axis, and Aj denotes the 
magnitude of correction therefor in the direction of a J axis. The 
magnitudes of correction for the position of a barycenter are 
calculated for each of the plurality of candidate images. 

The foregoing magnification k and the magnitudes of correction 
for the position of a barycenter (Ai,Aj) which are calculated for 
each candidate image are adapted to each candidate image . 
Consequently, the plurality of candidate images is transformed into 
images defined in the same apparatus of coordinates as the reference 
image. Thus, the transformation is achieved. Fig. 7 illustratively 
shows the result of the transformation. 

At step 210, mutual information shared by each candidate image 
and the reference image is calculated. Each candidate image to be 
subjected to this calculation is a transformed image. 

In order to calculate mutual information, the entropy h(a) of 
the reference image is calculated. The entropy h (b) of each 
candidate image is calculated. A joint entropy h(a,b) that is a 
combination of the entropy of each candidate image and the entropy 
of the reference image is determined. Using the entropies, mutual 
information is calculated as follows: 

Ml {a, b) = h(a) + h(b)-h(a,b) (12) 

The mutual information is determined for each of the plurality 
of candidate images. Since the plurality of candidate images is 
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transformed images, mutual information shared by each candidate 
image and the reference image can be determined under the same 
geometric conditions. 

Whether the amount of mutual information is large or small 
indicates a degree of approximation to which each candidate image 
approximates the reference image. Since the plurality of candidate 
images meets the same geometric conditions as the reference image , 
whether the amount of mutual information is large or small indicates 
a degree of approximation to which each candidate image approximates 
the reference image. Consequently, a candidate image sharing the 
largest amount of mutual information with the reference image is an 
image closest to the reference image. 

At step 212, a candidate image sharing the largest amount of 
mutual information is selected. Consequently, an image closest to 
the reference image is selected from among the candidate images. 
The image has the highest possibility that it may result from 
visualization of the same section of a patient body as the reference 
image does. Since the selected candidate image has already been 
transformed, registration has also been completed. 

Generally, the received images include an image that results 
from visualization of the same section of a patient body as the 
reference image does. Therefore, the desired image is automatically 
extracted through the aforesaid processing, and registration is 
automatically completed. 

Fig. 8 is a functional block diagram of the present apparatus 
performing the aforesaid actions. As illustrated, the present 
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apparatus includes a candidate extracting part 302, a transforming 
part 304, a mutual information calculating part 306, and an image 
selecting part 308. 

The candidate extracting part 302 extracts a candidate image 
similar to a reference image from among a plurality of received 
images. This functional block performs the processing equivalent to 
step 206. The candidate extracting part 302 is an example of an 
extracting means included in the present invention. 

The transforming part 304 transforms each candidate image. 
This functional block performs the processing equivalent to step 208 
The transforming part 304 is an example of a transforming means 
included in the present invention. 

The mutual information calculating part 306 calculates mutual 
information shared by each candidate image and the reference image. 
This functional block performs the processing equivalent to step 210 
The mutual information calculating part 306 is an example of a 
calculating means included in the present invention. 

The image selecting part 308 selects a candidate image that 
shares the largest amount of mutual information with the reference 
image. This functional block performs the processing' equivalent to 
step 212. The image selecting part 308 is an example of an image 
selecting means included in the present invention. 

Described above is a case where an image closest to an image 
produced by a PET apparatus is extracted from among a plurality of 
images produced by an X-ray CT apparatus. The present invention is 
not limited to the images produced by the X-ray CT apparatus and PET 
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apparatus but also applies to medical images produced by other 
modalities. Even in the case where the medical images produced by 
the other modalities are employed, an image produced by one modality 
and closest to an image produced by the other modality can be 
extracted in the same manner as mentioned above. Moreover, the 
present invention can apply not only to' two-dimensional images but 
also to three-dimensional images. Moreover, needless to say,, the 
images are not limited to the medical images. 

Many widely different embodiments of the invention may be 
configured without departing from the spirit and the scope of the 
present invention. It should be understood that the present 
invention is not limited to the specific embodiments described in 
the specification, except as defined in the appended claims. 
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